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AN EVALUATION OF CONTRACTOR PROJECTED 
AND ACTUAL COSTS 


Kent A. Kwiatkowski and Charles Buffalano 

Goddard Space Flight Center 


INTRODUCTION 

NASA contracts meeting certain category, dollar, and duration of performance criteria 
contain a requirement for contractors to submit monthly financial and manpower data. 

The NASA Form 533 shown in Figure 1 is the means through which these data are obtained. 

The contractor reports actual expenditures on a monthly basis, estimates of expenditures 
for the next four quarters, and the yearly total. These estimates are used at the Goddard 
Space Flight Center to track the contractor’s progress and to calculate the end of year 
accrued cost position of NASA flight projects. This document reviews the accuracy of 
these contractor estimates. 

For this study accuracy will be measured by the ratio of the actual expenditures in a 
quarter to the predicted expenditures for that quarter. It is important to understand that 
inaccuracy does not necessarily mean mismanagement or poor contractor analysis. At 
any point in time the contractor must estimate future costs based on the existing contract. 
Even if a contractor knows that the government is considering a major contract change, 
that change may not be reflected in the estimate until it is formalized. This means that 
the contractor does not even attempt to predict actual costs, but predicts the costs of the 
present contract. In an environment where total overruns of 200 percent are frequent, 
this distinction is more than academic. 

Since these estimates are used as part of the Center’s resource planning, several questions 
need to be asked. 

• What is the overall accuracy of these estimates? 

• Are projections made for the near future more accurate than those made further 
into the future? 

• How accurate are the yearly totals? 




Figure 1 . Quarterly contractor financial management report form 









• Is the accuracy of estimates made on service-, spacecraft-, or experiment-type 
contracts different? 

• Are projections made for smaller dollar amounts more accurate than those made 
for larger dollar amounts? 

• Is there any difference in the accuracy of estimates made for newer and for older 
projects? 

• Is there any difference in the accuracy of estimates made for application projects 
and for science projects? 

DATA ANALYSIS 

The contractor data in Goddard’s automated 533 system have been analyzed tor the 
fiscal years 1971 and 1972. There were a total of 204 contracts included in the data base 
and these represented 86 contractors. 

The approach used for this analysis was to compute the overrun ratio by dividing the 
quarterly projected cost into the quarterly actual cost. A ratio of 1.00 means the pro- 
jection was accurate; a ratio less than 1.00 means that actual costs were less than esti- 
mated; and a ratio greater than 1.00 means that actual costs were more than estimated. 

The statistics of these overrun ratios were then studied by plotting their distribution func- 
tions. For example. Figure 2 is the distribution of the quarterly overrun ratios for all pro- 
jects; it should be read as follows: Ten percent of the whole sample had overrun ratios 
less than 0.5, that is, they underran by a factor of 2 or more. Fifty percent of the sample 
either accurately estimated the quarterly projection or underran it, and 90 percent of the 
sample had overrun ratios less than 2.5. 

Another way to look at the situation is this: There is a 10 percent chance that the actual 
costs will be half or less than the estimated cost; there is a 50 percent chance that the 
actual cost will be equal to or less than the estimated cost; and there is a 90 percent chance 
that the actual cost will be less than 2.5 times the estimated cost. Since graphs are often 
difficult to read, the Appendix contains the distribution functions in tabular form. 

LIMITATIONS OF THE STUDY 

In later sections, conclusions will be drawn from the distribution functions for various 
subsets of the data. These conclusions should be considered tentative because the data is 
limited to a 2-yr period in one center’s history. This may or may not be typical enough 
to justify generalizing the conclusions. 

Furthermore, these statistics can only be used to adjust contractor estimates in the 
absence of specific information. If, for example, the life span of a 40-yr old adult male 
were to be estimated, the statistics in standard mortality tables might suffice. But if it 
were also known that he smoked 10 packs of cigarettes a day and had a family history of 
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cancer, it would be unwise to use the mortality tables. Judgement of the relevance of the 
additional facts would make the statistics inappropriate. 

THE EFFECTS OF TIME 

To decide if projections made for the near future are more accurate than those made for 
the more distant future, the ratios were grouped into five categories: 

• Estimates made 3 months ahead (nearest quarter) 

• Estimates made 6 months ahead (second nearest quarter) 

• Estimates made 9 months ahead (third nearest quarter) 

• Estimates made 1 2 months ahead (fourth nearest quarter) 

• Estimates made for the year’s total 

These distribution functions are plotted in Figure 3. The data show that the quarterly 
estimates become more accurate as they get closer in time, and that the contractor is 
better able to estimate and control annual expenditures than quarterly expenditures. 

Next it must be determined if these differences are statistically significant or if they are 
just due to chance. The Pearson chi-square test* was used to determine statistical signifi- 
cance for the differences of ratios among the various groupings. The result of this test 
had to be significant at the 0.05 level to be acceptable, meaning that there was a 95 
percent chance that ratios for a specific group were significantly different from the ratios 
of the total sample. In this case there is indeed a significant difference at the 0.05 
confidence level, between each of the four quarterly estimates and the yearly total. 

The graph shown in Figure 3 could be used to adjust a specific contractor estimate if its 
time range were known. Consider, for example, a 9-month estimate of $500K. Ninety 
percent of the time, the actual expenditure will not exceed S500K X 3.5 or S1750K. 

TYPE OF CONTRACT 

Contracts fell into three main categories: spacecraft, experiments, and service. The ratios 
of all 204 contracts were grouped into these three broad categories. The chi-square test 
was applied to each of the groupings and there was a significant difference at the 0.05 
confidence level. The distribution functions are shown in Figure 4. Differences are small 
but the lowest overrun ratios were found among the service contracts and the largest 
overrun ratios among the experiment contracts. Service contracts tend to be more 
definable, therefore less apt to change, so that costs can be more accurately estimated. 

On the other hand, experiments are probably the least definable of the three contract 
types, and are subject to the most change, resulting in the largest variance between actual 
costs and estimated costs. 


*Fisz, Marek, Probability Theory and Mathematical Statistics , John Wiley & Sons Inc., N. Y., 1963. 
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Figure 2. Fiscal year 71-72 data total sample. 



Figure 3. Fiscal year 71-72 533 estimates by time. 







DOLLAR EFFECTS 

To determine if projections made for smaller dollar amounts are more accurate than those 
made for larger dollar amounts, all 204 contracts were grouped into one of four 
categories: 

• Quarterly expenditures under $ 1 00K 

• Quarterly expenditures S100K but less than S500K 

• Quarterly expenditures S500K but less than $1,000K 

• Quarterly expenditures $1,000K and over 

The overrun ratios were computed, the chi-square test was applied to each group, and the 
resulting distribution function is shown in Figure 5. The test showed that there is a 
statistically significant difference between the cost ratios for the four groups of dollar 
values. 

Smaller contracts overran a little less than larger contracts, but between $ 100K and $ 1 
million seemed essentially the same. Prediction accuracy does not appear to be influenced 
greatly by the size of the contract. However, more absolute dollars are involved in larger 
contracts, so one hesitates to conclude where better management is exercised. 

PROJECT AGE EFFECTS 

The next question is to decide if there is any difference in the accuracy of estimates made 
for newer and for older projects. 

Of the 204 contracts, 1 1 5 were identified with a specific project and the other 89 were 
identified with a functional Goddard organization. The projects were classified as older 
or newer based on when Goddard manpower charges first appeared; projects with man- 
power charges prior to FY 1967 were considered older. Their ratios were computed and 
compared to the total sample via the chi-square test; the distribution functions are shown 
in Figure 6. 

There is no significant difference at the 0.5 confidence level between cost ratios of older 
projects and cost ratios of newer projects so the answer to the question appears to be no. 

PROGRAM AREA 

The final problem is to determine if there is any difference in the accuracy of estimates 
made for applications and science projects. 

Of the 204 contracts in the total sample, the 1 15 contracts that were identified with specific 
projects were grouped by application or science projects. The ratios were computed and 
compared to the total sample via the chi-square test, and the distribution functions are 
shown in Figure 7. There is a significant difference at the 0.05 confidence level between 
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the overrun ratios of application projects and the cost ratios of science projects. One 
reason for the higher rates for application projects might be that during the sample period 
there was a push on the application projects because of tight launch schedules that may 
have resulted in less detailed definition. Science projects developed at a slower rate and 
therefore were capable of more detailed definition. 

CONCLUSIONS 

The analysis of contractor 533 reports showed that the overruns of projected costs are: 

• Smallest for the nearest quarter projected and larger for projections further into 
the future 

• Smallest for service and largest for experiment contracts 

• Smallest for expenditures under $ 1 00K in a quarter 

• Not significantly different for newer projects as compared to older projects 

• Smaller for science projects than application projects 


Goddard Space Flight Center 

National Aeronautics and Space Administration 
Greenbelt , Maryland August 13,1973 
039 - 03 - 43 - 13-51 


9 



APPENDIX 

DISTRIBUTION TABLES 


This Appendix contains the distribution functions in tabular form for contractor data in 
Goddard’s automated 533 system for the fiscal years 1971 and 1972. 


Table 1 

Ratios for Total Data Base 
by Nearness of Projection. 



Time Periods 


Percent 

3 Months 

6 Months 

9 Months" 

1 2 Months 

Total 

100. 

99.33 

43.72 

65.00 

35.31 

99.33 

90 

1.61 

2.53 

3.73 

4.65 

2.51 

80 

1.22 

1.80 

2.28 

3.20 

1.69 

70 

1.07 

1.45 

1.77 

2.34 

1.33 

60 

0.99 

1.24 

1.43 

1.79 

1.13 

50 

0.93 

1.08 

1.20 

1.37 

1.00 

40 

0.85 

0.95 

1.08 

1.15 

0.92 

30 

0.75 

0.85 

0.94 

1.00 

0.82 

20 

0.64 

0.68 

0.79 

0.79 

0.68 

10 

0.45 

0.50 

0.54 

0.50 

0.49 

0 

0.02 

0.03 

0.03 

0.03 

0.02 

Sample 

Size 

1015 

562 

376 

239 

2197 
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Table 2 


Ratios for Service Contracts 
by Nearness of Projection. 



Time Periods 


Percent 

3 Months 

6 Months 9 Months 

1 2 Months 

Total 





Sample 

Size 
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Table 4 


Ratios for Experiments Contracts 
by Nearness of Projection. 


Percent 

Time ] 

Periods 

Total 

3 Months 

6 Months 

9 Months 

1 2 Months 

100 

8.00 

43.00 

65.00 

31.50 

65.00 

90 

1.54 

2.88 

4.51 

4.66 

2.81 

80 

1.21 

1.94 

2.73 

3.50 

1.81 

70 

1.06 

1.61 

2.00 

2.54 

1.41 

60 

0.96 

. 1.33 

1.51 

2.01 

1.16 

50 

0.87 

1.14 

1.22 

1.47 

1.00 

40 

0.78 

0.99 

1.08 

1.11 

0.89 

30 

0.69 

0.84 

0.90 

0.97 

0.77 

20 

0.56 

0.63 

0.70 

0.79 

0.61 

10 

0.40 

0.47 

0.50 

0.46 

0.45 

0 

0.02 

0.03 

0.03 

0.03 

0.02 

Sample 

Size 

580 

363 

243 

153 

1339 


Table 5 

Ratios for Contracts with Less Than $ 1 00K Expenditures 
in Any One Quarter by Nearness of Projection. 


Percent 

Time Periods 

Total 

3 Months 

6 Months 

9 Months 

1 2 Months 

100 

8.00 

31.50 

31.50 

31.50 

31.50 

90 

1.60 

2.49 

3.50 

3.55 

2.25 

80 

1.21 

1.64 

2.22 

2.41 

1.52 

70 

1.06 

1.32 

1.50 

2.15 

1.20 

60 

0.96 

1.09 

1.22 

1.41 

1.03 

50 

0.85 

0.90 

1.08 

1.05 

0.91 

40 

0.75 

0.73 

0.89 

0.94 

0.76 

30 

0.64 

0.61 

0.66 

0.65 

0.63 

20 

0.50 

0.47 

0.51 

0.50 

0.50 

10 

0.34 

0.35 

0.37 

0.28 

0.35 

0 

0.02 

0.03 

0.03 

0.03 

0.02 

Sample 

Size 

514 

249 

160 

101 

1024 
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Table 6 


Ratios For Contracts With $100- to S500-K Expenditures 
in Any One Quarter by Nearness of Projection. 


Percent 

Time Periods 


3 Months 

6 Months 

9 Months 

1 2 Months 

100 

99.33 


65.00 

22.33 


90 

1.61 

2.81 

3.84 

4.02 

2.76 

80 


1.89 

2.25 

2.81 

1.78 

70 


1.51 

1.75 

2.28 

1.37 

60 


1.33 

1.45 

1.81 

1.16 

50 

0.96 

1.14 

1.21 

1.44 

1.04 

40 



1.09 

1.20 

0.97 

30 

0.83 


1.01 

1.06 

0.90 

20 

0.75 

0.87 

0.89 

0.88 

0.81 

10 


0.72 

0.78 

0.78 

0.67 

0 

0.21 

0.48 

0.41 

0.39 

0.21 

Sample 

Size 

332 

207 

137 

82 

758 


Table 7 

Ratios for Contracts with $500- to $1 ,000-K Expenditures 
in Any One Quarter by Nearness of Projection. 


Percent 

Time ] 

Periods 

Total 

3 Months 

6 Months 

9 Months 


100 

4.02 

2.82 


■ 

13.02 


2.05 

1.97 



3.03 

80 

1.31 

1.69 



1.98 

■ 

1.13 

1.51 



1.64 

■ 

1.08 

1.42 



1.28 


1.00 

1.29 



1.16 


0.99 

1.22 



1.04 


0.94 

1.02 





0.89 

0.88 



0.94 

10 

0.71 

0.72 



0.80 

0 

0.55 

0.67 



0.55 

Sample 

Size 

37 

13 

8 

5 

63 
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Table 8 


Ratios for Contracts with $ 1 ,000K Plus Expenditures 
in Any One Quarter by Nearness of Projection. 


Percent 

Time 1 

Periods 

Total 

3 Months 

6 Months 

9 Months 

1 2 Months 

100 

13.89 

43.72 

46.62 

35.31 

46.62 

90 

1.42 

2.24 

3.26 

5.07 

3.02 

80 

1.21 

1.86 

2.49 

3.95 

1.90 

70 

1.04 

1.51 

1.97 

2.90 

1.45 

60 

0.99 

1.33 

1.65 

1.95 

1.26 

50 

0.97 

1.19 

1.44 

1.64 

1.12 

40 

0.92 

1.05 

1.23 

1.32 

1.00 

30 

0.86 

0.94 

1.10 

1.21 

0.94 

20 

0.78 

0.89 

0.96 

1.05 

0.88 

10 

0.66 

0.84 

0.89 

0.91 

0.77 

0 

0.50 

0.56 

0.59 

0.60 

0.50 

Sample 

Size 

132 

93 

71 

51 

347 


1'able 9 

Ratios for Contracts Associated with 
Newer and Older Projects. 


Sample 
Size 


1 

Percent 

Newer 

Projects 

Older 

Projects 

100 

7.03 

99.33 

90 

1.97 

2.48 

80 

1.44 

1.62 

70 

1.21 

1.30 

60 

1.03 

1.11 

50 

0.96 

1.00 

40 

0.90 

0.88 

30 

0.78 

0.77 

20 

0.66 

0.61 

10 

0.45 

0.45 

0 

0.10 

0.02 


108 

1286 
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Table 10 


Ratios for Contracts Associated with 
Science and Application Projects. 


Percent 

Science 

Projects 

Application 

Projects 

100 

22.33 

99.33 

90 

1.88 

3.02 

80 

1.38 

2.00 

70 

1.16 

1.48 

60 

1.04 

1.24 

50 

0.95 

1.04 

40 

0.85 

0.93 

30 

0.73 

0.80 

20 

0.57 

0.69 

10 

0.44 

0.50 

0 

0.02 

0.02 

Sample 

Size 

735 

659 




